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ABSTRACT i 

The lack of high quality data to describe and monitor 
U.S. educational systems hampers reform decisions and will impair 
reform evaluation. This paper sets forth the Council of Chief State 
School Officers* (CCSSO) recommendations for the Council and federal, 
state, and local education agencies to help improve .the quality of 
educational indicators. The CCSSO will assume a national leadership 
role in the formulation and coordination of educational assessment 
policies. Recommendations to the U.S. Department of Education 
include: (1) integrating evaluation project reports; (2) revising 
"State Education Statistics" (popularly known as Secretary Bell's 
"Wall Chart"); (3) providing financial aids for state assessment 
programs; and (4) continuing joint federal/state/local planning, 
is recommended that each state: (1) develop a systematic approach to 
monitoring education in relationship to state goals and context; (2) 
establish high standards for its collection and presentation of 
outcome indicators; and (3) work with other states to establish a 
core set of indicators for across-st&te comparisons. (BS) 
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EDUCATION EVALUATION AND ASSESSMENT XM THE UNITED STATES 
Introduction 

During the put two years there has been an Intense and productive 
nationwide debate about the condition of education in the united states 
and how to Improve it. The consequences of this debate have touched 
almost every community in the nation as the several states have carried 
out reviews of their policies and p ro pos e d significant nee re fores. 
However, as citizens' groups, legislators, governors, and local and state 
boards of education have examined educational problems, they have often 
been frustrated by the lack of good information about the condition of 
education. 



High quality data are essential to the continuing eomentue of education 
reform. The publication of data brings education to the attention of the 
public and aalcee it an issue of enduring concern. By comparing the 
performance of a school, district, or state with itself over tise, or with 
other schools, districts, or states, data serve to exhort, r motivate, or 
reward . 

i. 

A second use is to aid local, stats, and national policymakers in 
understanding the consequences of changes in policy and to aid them in 
implementing policies once they are adopted. Because of the variety that 
exists across the nation's school districts, numerous natural experiments 
are continually taxing place as academic standards are increased, steps 
are taken to enhance the quality of teaching, the school day or year is 
lengthened, or the level of resources available for educating specific 
groups of students changes. By observing the impact that these changes 
have on the outcomes of schooling, poUcymaXers can estimate the costs and 
benefits of pursuing alternative strategies for improving student 
performance. 

Finally, and coming full circle, data serve to make education accountable 
to the public. At no time have more citizens been involved at all levels, 
studying the problems of education and proposing solutions. In easy 
states and localities, significant nee dollars have been made available to 
fund these proposals, with this commitment comes a concomitant interest 
in the assessment and evaluation of the system and the systematic 
reporting of its strengths and weaknesses, successes, and failures. 

examples of the weakness of present educational information abound. Sere 
are but twos 

1. The National Science Foundation (BSD reported a major 
national problem with the supply of qualified high school teachers of 
mathematics and science. Shortly afterward, the c*neraJ_A5*otinTing office 
found that data supporting NST's conclusion were too weak as a bas%s for 
reasonable policy, and the problem, if there was one, was limited to^a few 
regions rather than the entire nation. ^ , 

2. The U.S. Department of Education published a statistical 
chart in 1984 which contained the -most comprehensive data available on 
the performance of state educational systems.- The purpose was to compare 
etates so that state leaders could "better consider appropriate reforms. - 
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The only muuzm of student achievement available Cor all states vers the 
* college entrance examinations— the Scholastic aptitude Test (SAT) and the 

American College entrance Examination (ACT). These scores are not 
representative of the quality of the states* educational systems. They do 
not reflect the diversity of the students or programs. They do not assess 
the quality of the elementary or middle school. At best they represent an 
assessment of the aptitude of college going twelfth graders. 

The lack of high quality data to describe and monitor educational systems 
in the U.S. hampers reform decisions and will Impair attempts to evaluate 
the effects of the reforms, we need valid ways of assessing the quality 
of education on a continuing basis. 

Steps are being taken to Improve assessment and evaluation information to 
meet this problem. At the federal level, the U.S. Department of E duca tion 
has asked the National Academy of Sciences to examine the federal systems 
of data collection and analysis. The Department has also established a 
Federal Indicators Project in collaboration with nationally base! state 
organizations to define the key statistical Indicators that need to -be 
collected by the federal government. 

^ " These are important, but not sufficient, steps to deal with problems that 

affect the entire educational system, from the federal level to the 
classroom. The Council of Chief State School Officers has worked over the 
past year to develop a set of r e comm en dations for the Council and federal, 
state, and local education agencies, designed to help Improve the quality 
of educational Indicators. This paper sets forth the recommendations. 

National c oncern s 

Since the birth of the Office of Education in 1867, the collection and 

dissemination of national statistics on education has been a federal 

responsibility. The federal government uses statistics to assess progress 
toward federal goals jsuch as the provision of opportunities for the 
handicapped, to track federal funds to ensure that their use me*-"* the 
intent of congress, to assess compliance with civil rights stat r s, and 
to monitor the health of the educational system in general. Thin last 
purpose leads to the collection of large amounts of data describing the 
financing of the system and the quantity of resources used In the 
schools. Less complete data are gathered on the quality of the resources 
and the outcomes of the educational process, yearly, the Department 
assembles a three foot stack of reports containing thousands of 
statistics. Although many of the federal reports are well done and useful 
to both policy makers and researchers, they do not provide a good base for 
policy making. 

To some extent, this problem exists because a variety of different 
agencies within the federal govern m e nt produce statistical reports, 
including agencies both within and outside of the Department of 
Education. Better coordination across agencies can address this aspect of 
the problem. To a greater extent, however » the problem exists because 
there is little or •*? org«u»izir*% theory or sodel to guitfa the collection 
and presentation of educational data. 




The Federal Indicators Project, and the national Academy Study show promise 
for improving the quality and use of federally collected data on 
education. Equally important is the federal sponsorship of research 
efforts to improve the quality of educational assessment and evaluation. 
Of special importance is research work on the measurement of student 
achievement, both for use by teachers in the classroom and for policy 
makers interested in aggregate measures of achievement at the school, 
district, state, and federal levels. 

The need for these efforts is strongly supported by even a cursory 
analysis of Secretary Bell's 1984 "vail chart- entitled -State Education 
Statistics.- The chart contains v only two student -outcome- measures 
(graduation rate and college entrance examination test score seans) and 
each has very limited value. There is no measure of teacher quality or of 
the quality of the curriculum, there is no eeasure of the opportunity for 
students to take academic courses. The quality of the available data «aa 
Influenced by the Secretary's decision to contrast states; this required 
that there must \ be common data available from each state. But even if the 
Secretary had focused only on nationwide data, the results would have been 
unsatisfactory. Of the problems listed above, only the limits of 
achievement test information would have been even partially 



The publication of the -well chart- also brought to the forefront the 
issue of state-to-atate comparisons, for years, comparisons among states 
on a wide variety of education measures have existed within the pages of 
Department of Education reports. These reports have even included 
rankings of the states on such eeasuree as par pupil expenditure and 
teacher/pupil ratio, national attention to the -wall chart- was attracted 
by two things t first, the secretary held a press conference and presented 
the chart, which was perceived by the press as a -report card- for the 
states* and second, the chart highlighted student outcomes and ranked 
states on them, thereby giving the media off every state a local story. 

on a technical level, state-to-state comparisons are problematical for a 
variety of reasons beyond those given above concerning the quality of the 
data. The most important reason is that states differ greatly in the 
characteristics of their populations, the levels of available resources, 
and the curricula taught in their schools. These differences are 
particularly critical when student outcomes are being compared . Moreover, 
state comparisons on student achievement tests invoke the spectre of a 
single national exam, a vision that challenges the U.S. tradition of state 
and local control over education. On a political level, however, the 
attention given to the Secretary's -wall chart- makes inevitable future 
state to state comparisons on outcome a 



These are important matter* for the future of our nation's educational 
system. Th task now is to make sure that the data are as high quality as 
possible ana given appropriate interpretation. This ha* both short- and 
long-term implications. In the short run, the Secretary's -wall chart" 
for 1985, if there is one, must be different from the 1984 version. Over 
the long-term, the Federal Indicators Project, in conjunction with state 
and local group*, should provide a structure to improve the quality of 
data critical to federal concerns. This effort should result in 
standardized and Improved specifications for gathering and reporting much 
of these data. 
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Whether such an effort should include federal specification of a set of 
indicators that would be collected by every state is another natter. The 
federal gove r nm en t can collect data that represent 4 the status of the „ 
nation on core educational issues without having the data be 
representative' at the stats level. Yet, if we accept the argument that 
state-to-state comparisons are bound to happen, it is important to ensure 
that the comparison* be as policy relevant and as valid and fair as , 
possible. An alternative to federal specification would be for the states 
collectively to provide the leadership. This would offer balan ce t o the 
potential centralization of policy implicit in the "federal government's 
taking the initiative. For much of the data to be collected, the eventual 
difference would probably be small. It might even be small on the most 
sensitive) of issues i the comparison of states on achievement test 
results. ♦The process for determining the policies in a democratic and 
decentralized system, however, is critical to the success of the policies. 

r , 

A final nationwide issue needs to be considered. The current debates 
about education make much of our nation's economic and educational 
position vis a vis other nations. The natural experiment made possible by 
the great variations among nations in educational policies indicates that 
we might learn much by examining the differences among nations. Yet 
remarkably little systematic data are gather** on which to make legitimate 
comparisons among nations. The single international organization 
established for this purpose (the International Educational Achievement 
organization — EEA) has received only sporadic support from the U.S. 
Government. It is time to address this issue. 

stats and Local Conoerms 

Arguments about the need for the nation to have a valid set of indicators 
collected over time, also hold at the stats level. These indicators may 
be somewhat different from those viewed as necessary by the federal 
government. Each state say' value certain outcomes differently, and no 
group of analysts has the -"right" model. These differences, however,, do 
not diminish the need for a state to be clear about the outcomes important 
to it or about the way that it assesses the effectiveness of resource use 
within its boundaries. 

This point is particularly Important now as the several states embark on 
programs of school reform. The quality of a state's efforts to evaluate 
the progress of its reforms will depend upon the quality of its 
educational data bass. A solid data bass containing regularly gathered 
measures of resources and outcomes will provide a foundation for 
monitoring the reforms. In the short-run. for example, the effects of 
many reforms on student achievement outcomes may not be noticeable, 
instead they may be seen in such areas as increased access to academic 
courses or to dropout prevention programs. States with measures only of 
student achievement -will know little about these issues. Similarly, 
states with only narrow achievement measures, as in some minimum 
competency examinations, will not be able to detect the effects^ of reforms 
like increased academic requirements, which might have mors impact on high 
achieving students. 
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Additionally, because comparisons of states will occur with increasing 
frequency, it i» important for the states to build the capacity to learn 
from these comparisons. Por a state considering whether to increase its 
high school graduation requirements, the experience of similar states that 
have recently carried out such a policy change could be eery informative. 
For this purpose it would be particularly useful for each state to have a 
common core of data that would enable valid and useful comparisons and a 
capacity to connect stats data to nationwide information. These data, of 
course, could be collected on samples of schools and students, reducing 
the extra burden on the 



rinally, the process of assessing and monitoring educational progress is 
important at the ^Local level. Measures of outcome should be clearly 
related to the purposes and goals of the system, and there should be 
better- ways of assessing the quality of resources in local education 
agencies and schools. In these ways the federal, stats, and local needs 
are similar. There is still another need at the local level, however, 
which cannot be met by measures of institutional effectiveness. This is 
the need for ways of diagnosing the needs and directing the education of 
individual students. The theory, methods of construction, and use of 
these measures are qualitatively different from that of the outcome or 
achievement measures used for evaluating federal, state, and local 
systems. More needs to be done in the development of these measures, for 
they may be the most important of all. 

/ 

This paper raises critical issues regarding educational assessment and 
evaluation and leads to recommendations for the OCSSO. These 
recommendations are addressed; to the OCSSO itself and to federal, state 
and local governments. They cover a broad range of assessment and 
evaluation issues with a partlclar focus on the role that the OCSSO eight 
take in providing direction and leadership over the^next several years. 

At this Wme of widespread reform in the united States, the OCSSO has an 
important responsibility for leadership in the development, use, and 
interpretation of assessment and evaluation procedures and results. 

Education, assessment and evaluation take place at different levels t the 
individual stud int, the classroom, school, school district, state, nation, 
and world. A' major .challenge is to design assessment and evaluation 
procedures which will allow for efficient use of data for decision making 
at each level and, where appropriate, for use at multiple levels. Beyond 
the longer-range concern of tracking the health of the U.S. educational 
system to provide data for informed policy making, the task of developing 
models and implementing testing and data collection systems is of great 
ismedlate concern to the OCSSO, for without adequate data we will not be 
able to monitor and assess the impact of the various reforms. The 
recomnendatlons that follow are directed to the OCSSO, to the U.S. 
Department of Education (and those responsible for federal 
appropriations — the U.S. congress and the President), to the states 
individually, and to local education agencies. 
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AMUDt a MtlflMJ leadership cola In thf formulation and 
coordination of policies regarding the assessment of the K-12 
educational systea. Work actively with federal and state agencies 
to improve their capacity to gather, analyse,' and report on a 
variety of assessment matters. 

The Committee on Coordinating Educatlonal*Xnformatlon and Ressearch 
(CUR) of the CCSSO should monitor the development of state and 
national policies. St might also develop model policies for states 

establish In the CCSSO a coordination center on assessment and 
evaluation proga wins of the states. The center would provide states 
with timely information on practices In other states i support 
cooperative efforts among states to align their programs mora 
closely with one another; and promote coordination among 
international,, national, and state assessment programs. The center 
shall be funded by the Council and/or outside sources. The scope 
of the work for the center and appropriate work schedule shall be 
prepared by the CEIR committee and presented to the Council 
Membership at the March 1985 Legislative Meeting. 



work actively with the National center Cor Education Statistics 
(MCE3) to ensure that reporting of data from all sources is 
accurate and timely. 

work wit)|the major test publishers, survey organizations, 
corporations, and other groups that have large-scale assessment 
programs, to examine methods for developing state-by-state 
comparative data that are valid. 

work to develop interim measures of institutional improvement to 
■set the public need for short-term data on the effects of recent 
educational reforms. Examples of such measures arms 

a. the Increase in the number of academic courses taken by 
students i u 

b. the increase in the number of science teachers taking summer 
training i and 

c. the increase In the number of students in dropout prevention 
prograns. 

U.S. Department of Education (EP) 

Integrate the projects through which the Secretary reports on the 
state of Education in the nation by combining reports — Sbs. 
rendition o € Education. Education Indicators Project, -wall Chart," 
and other statistical and asjessment reports. Request increased 
appropriations for the Department of Education (Rational Center for 
Education Statistics) assessment and evaluation efforts by five to 
six times current . level (currently sa+ million per year) to make 
the capability comparable with national reporting in health, 
agriculture, and other functions. 
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over thn short-run, in cooperation with the CCSSO, revise -State 
Education statistics i Stat* Performance outcomes. Resource Inputs, 
and Population Characteristics, 1972, and 1982" prepared by the 
o.S. oepartsent of Education, and released in January 1984 
(popularly known as the secretary's "wall Chart") so that ifci a) 
focuses on nationwide indices and trends i b) represents any state 
performance or outcome through trends or net change for the state 
itself rather than by rank order of states t c) includes Scholastic 
aptitude Test (SAT) or American college Entrance Examination (ACT) 
scores only if reported along with the state percentages of twelfth 
graders represented by the SAT/ACT scores i d) includes student 
retention rates only if adjusted by net In/out migration and if 
data from different states are comparable j and e) includes the 
number of G£D's awarded in the state and the change over the 
years. (It is important to recognise that post hoc analyses can 
'never adequately adjust measures to ensure comparability. If 
' comparisons are desired, they must be carefully planned in advance 
so that the indicators can be defined and weighted according to the 
questions they are being asked to answer. ) 

Integrate education assessment and evaluation results and other 
pertinent information from other federal agencies such as the 
Department of Defense (i.e., entrance test results). Department of 
Labor, National Science Foundation, and Census Bureau into the 
Education Department assessment and evaluation program. 

Provide financial aid to states to stimulate new state assessment 
programs and strengthen existing stats assessment efforts. Federal 
aid might also be used to relate state- programs to the National 
Assessment of Educational Progress (NAEP) or other nationwide 
testing programs. 

To the rcyy * and U.S. Department^ of Education Together 

1. continue Joint federal/state/local planning efforts on assessment 
and evaluation. The efforts should include indicator model design 
and identification and use of joint opportunities Cor data 
gathering, analysis, and reporting. This includes examination of 
indices in other fields such as health, economics (CPI, Dow Ames 
Average), and welfare; and the development of indices for student 
achievement and for the relative challenge presented by differing 
groups of students (e.g., different soci o e c o n o m ic groupings) to 
meet the same achievement levels. 

2. Establish a long-term commitment to develop a core set of education 
indicators and an accompanying analytical model which accurately 
portrays the educational system and effectively measures 
educational effectiveness. This effort will require two or three 
years, at least. 

3. Educate the public, policy-makers, educators, and the media on the 
appropriate uses of data, assessment results, 'and education 
indicators, and the inferences that may legitimately be drawn from 
them. 
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4. Develop iiHMMnt and evaluation exchange programs with other 

nations and support united States participation in International or 
cross-national education assesssent projects. Consider systematic 
support of the International Educational Achievement (ISA) effort. 
Also examine opportunities to work with the Organisation of . 
Economic and Community Development (GCCD). 

10 

1. Each state should develop a systematic approach to monitor j-g the 
status of education in relationship to state goals and contuse, 
specif lcallyi 

a. A set of outcomes for the educational system and a way of 
measuring them should be agreed upon by various interests 
Within each state. 

b. Each state should develop a c on ce ptu al model which links 
outcomes to indicators critical to the quality of education. 
One approach to defining input indicators is to break them . 
into four categories t 1) measures of Che quality and level 
of resources such as teachers and curriculum materials i 2) 
measures of the proces^ts within the schools such as time 
spent on instruction, teaching strategies, and in-service 
training approaches i S) measures of the characteristics and' 
needs of the students such as the percentage of handicapped 
and limited English-speaking ability i and 4) measures of the 
policy environment Including the rules, regulations, and 
standards governing the state's educational system. The 
measures selected as input Indicators should: all have a solid 
and stable relationship to one or more of the ou tco m e 
indicators. 

c. Each state should publish a yearly report which contains 
displays and analyses of cross-sectional trend data showing 
the condition of its educational system as represented by 
outcomes and other indicators. 

2. Each state should establish high standards to be followed in its 
collection and presentation of outcome indicators i 

a. Multiple indicators should be presented (e.gs, achievement 

a scores, dropout rates, numbers of high school equivalencies, 
college-going rate, percentage of 18-19 year old youth in 
jobs and in the armed forces). 

b. Data should be presented over time as trend and gain scores. 

c. Work should be carried out to develop a "challenge index** 
which would give an indication of the gain of the state 
relative to its starting point. 

d. Presentation of data should include ethnic, socio-economic, 
language dominance, and handicapped groupings. 

To states t Rmccmmendatlane Regarding Indicators Which can be 



Implementation of the following five recommendations shall be in 
accordance with a plan prepared by the CEIR Committee and presented to the 
Council at the Annual Meeting Novembe r 1985 for approval. 
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The various tutM also should work with each other and with the 
GCSSO to establish a "core set" of input and outcome indicators to 
ha collect ad on a yearly basis by each state. This would 
facilitate the examination of the effects of state policies by 
making comparisons available on a etate-to-etate and 
cluster of 1 states basis. 

e. The format and definition for each en the indicators would be 

common to all. the states, 
to. These indicators would not necessarily replace existing data 

gathered by the state i rather, they should be viewed as 

augmenting an existing data base. 

c. The CCSSO and the states should work to make the indicators 
as parsimonious and inexpensive to collect as possible. 

d. The indicators should be published by each state along with 
its other indicators in a yearly status report. 

e. These indicators could be collected bit a sample basis within 
states rather than for the total population co n ce rn e d . This 
would decrease cost and burden and would sake it clear that 
the indicators ware not replacing existing data collection 
efforts. 

(The implementation of this recommendation would take place over a 
two-year period and would be coordinated by the CCSSO. ) 

States should work with each other and with the CCSSO to establish 
a "core set- of outcome indicators. There should be multiple 
indicators including data from educational achievement teste. The 
ccsso will sponsor a planning study to develop specifications and 
criteria for defining these outcome indicators. 

With respect to achievement testing, the planning study should 
develop criteria for states to use in their selection of 
instruments for assessing student achievement. Over a period of 
approximately five years, the Implementation of these criteria 
would result in states testing a common set ot grades and a common 
set of skill and content areas with a set of teats that would be 
equa table across; the states. 

In addition, the achievement tests meeting the criteria should have 
a variety of special characteristics. They should i a.) assess 
basic and higher order conceptual skills i b. ) assess a core of 
content knowledge* in science, social studies, and English »* well 
as in mathematics and reading; c. ) provide the opportunity for a 
writing sample and a -production" item in some domain other than 
English i and d. ) be of the highest possible quality and be 

continually upgraded as knowledge Improves. 

« 

Given the current level c " resource allocation and technical 
expertise available, developing and maintaining high quality 
testing instruments will be difficult. States will need to augment 
significantly the resources devoted to testing if this goal Is to 
be set. 
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5. The "ooit - tftt(i) should augment rather than replace the existing 
state aasessnents c*, achievement. ' The primary function of the 
-core" tests would toe to provide data representative of a state's 
progress and level of educational performance. Therefore, the test 
■ay be administered to samples rather than total populations of 
students at the selected grade levels, samples per grade level can 
be quite small and provide reliable stats level information. Zn 
addition thexf. are a number of legitimate p s ycho m e tric strategies 
which may be used to limit the testing time for any Individual 
student. It would be straightforward to design a procedure that 
tested approximately 400 classrooms, each for one hour apiece at 
each of three grade levels in a state and to come up with very 
reliable and extensive data. This would not generate data 
representative at the local level. local districts could be given 
the option to augment the data collection to establish their own 
representative sample. 

To Local location mgenclSJi 

we acknowledge that . . ^commendations made to local districts should be 
given more attenti n follow-up documents. Much of the burden in the 
proposed activities /ill fall to them, and the recommendations directed 
toward them could be significantly enhanced. 

l. The CCSSO encourages local education agencies (LEAs) to make 

effective use of diagnostic and evaluative Instruments to Improve 
the quality of instruction in their schools. The CCSSO will hold a 
conference on the use of diagnostic test instruments which will 
assess the state of the art and, it 1c hoped, provide useful 
information for local districts. 

3. The CCSSO encourage local districts to establish systematic data 

collection approaches in order to provide timely and accurate data 
to concerned citizens and policy makers. These data should Include 
outcome indicators (such as achievement, retention, and 
college-going rates) as well as input indicators of the sort 
suggested for the states. 
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